Skip to content

Use the true end of the buffer in serialize() with a header - #593

Merged
proost merged 3 commits into
masterfrom
serialize-end-ptr
Nov 21, 2026
Merged

proost merged 3 commits into
masterfrom
serialize-end-ptr

Conversation

@leerho

@leerho leerho commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #591, which fixed this for EBPPS. The same line is in seven more sketches: var_opt, tuple, KLL, density, REQ, frequent items and classic quantiles. serialize(header_size_bytes) set end_ptr = ptr + size, where ptr is already past the header and size includes it, so the end was header_size_bytes past the buffer.

What that did:

  • The capacity checks of the serde were given that end, so a serde that writes more than its size_of_item() reported could write past the buffer instead of throwing std::out_of_range.
  • Density compares its output with that end, so serialize() threw whenever a header size was given.

While testing this I found that REQ's get_serialized_size_bytes() counted a compactor header for 2 to 4 items, which are written as raw items without one. serialize() to bytes therefore returned 20 trailing zero bytes that the stream form doesn't have. The size now matches what is written. Deserializing the longer images still works, since the trailing bytes are ignored. The cross-language snapshots are written with the stream form, so they don't change.

Tests:

  • For each of the seven sketches: serialize with a header and deserialize the bytes after it.
  • For the six with a serde: a serde that reports each item one byte shorter than it writes must be stopped by its capacity check. A test helper for this, undersized_string_serde, is added to common/test. On master these tests overflow the buffer under ASan.
  • REQ: the byte size, the stream size and get_serialized_size_bytes() agree for 1 to 6 items.

Frequent items: both serialize() overloads copied the items into a raw allocation for the serde and freed it only after the serde returned, so a serde (or an item copy) that threw leaked the copies. The new header test made the serde throw, and LeakSanitizer in CI caught it (0a8b021). The copies are now held in vectors; the bytes are unchanged. A new test with test_allocator and a serde that always throws checks both overloads.

All new tests fail on master and pass here. All tests pass under ASan and UBSan.

🤖 Generated with Claude Code

serialize(header_size_bytes) set end_ptr to ptr + size, where ptr is
already past the header and size includes it, so the end was
header_size_bytes past the buffer. The same as #591 fixed for EBPPS, in
var_opt, tuple, KLL, density, REQ, frequent items and classic quantiles:
- The capacity checks of the serde used that end, so a serde that wrote
  more than size_of_item() reported could write past the buffer.
- Density compared its output with that end and threw whenever a header
  size was given.

REQ: get_serialized_size_bytes() counted a compactor header for 2 to 4
items, which are written as raw items without one, so serialize() to
bytes returned 20 trailing zero bytes that the stream form doesn't have.

Tests: serialize with a header and deserialize the bytes after it, and
check that a serde that under-reports the size of its items is stopped
by its capacity check. A test helper, undersized_string_serde, is added
to common/test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested a review from tisonkun November 20, 2026 18:29
@coveralls

coveralls commented Nov 20, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 44818621979

Coverage increased (+0.03%) to 97.078%

Details

  • Coverage increased (+0.03%) from the base build.
  • Patch coverage: 32 of 32 lines across 7 files are fully covered (100%).
  • 2 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

2 previously-covered lines in 1 file lost coverage.

File Lines Losing Coverage Coverage
common/include/serde.hpp 2 98.02%

Coverage Stats

Coverage Status
Relevant Lines: 17454
Covered Lines: 16944
Line Coverage: 97.08%
Coverage Strength: 1885270.49 hits per line

💛 - Coveralls

Both serialize() overloads copied the items and weights into raw
allocations for the serde, and freed them only after it returned, so a
serde or copy that threw leaked them. The new header test in this PR
made the serde throw and LeakSanitizer on Linux caught it. The copies
are now held in vectors. The serialized bytes are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A purge can clear every counter while the sketch stays non-empty.
serialize() then reserved zero-length vectors and passed their null
data() to memcpy, which UBSan reports on Linux in "purge clearing all
counters serialized form".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@proost
proost merged commit 1e49055 into master Nov 21, 2026
35 checks passed
@proost
proost deleted the serialize-end-ptr branch November 21, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants